Method and apparatus for segmenting biological cells in a picture

ABSTRACT

A method of segmenting biological cells in a picture so that the biological cells represent a foreground of the picture includes a step of applying a first fast marching algorithm to the picture or to a pre-processed version of same in order to obtain a first fast marching image. In addition, the method includes a step of segmenting the first fast marching image or a further-processed version of same into a plurality of homogeneous regions. Furthermore, the method includes a step of mapping each of the homogeneous regions to one node of a graph, respectively. In addition, the method includes a step of classifying each homogeneous region either as background or foreground on the basis of the graph. Moreover, the method includes a step of applying a second fast marching algorithm within the homogeneous regions classified as foreground so as to segment the foreground into individual biological cells.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to German Patent Application No.102010024859.2 filed 24 Jun. 2010, which is incorporated herein byreference.

Embodiments of the present invention relate to a method and an apparatusfor segmenting biological cells in a picture, as may be used, forexample, for detecting and segmenting leukocytes in blood smears or bonemarrow smears.

BACKGROUND OF THE INVENTION

One important component of hematology is the differential blood count.Systems from the field of “computer-assisted microscopy” (CAM) enableautomatic analysis of blood smears and support hematologists inclassifying cells, and thus they form a supplement for modernhematological laboratory diagnostics. It is in a fast, precise andhighly efficient manner that modern hematology systems provide importantinformation about the cell population of peripheral blood. However, upto 40% of samples may subsequently be manually differentiated under themicroscope in clinics and laboratories. Specifically this last step maybe accelerated and objectified by means of a system of“computer-assisted microscopy”, as is shown in FIG. 11. This leads toboth reducing the amount of work involved and increasing the quality ofthe results. On the basis of innovative concepts of image processing,leukocytes in the blood smear are localized and classified intoclinically relevant subclasses. Reference data sets which are classifiedin advance by experts and may be extended any time serve as the basisfor said classification.

A typically CAM system (as is shown by way of example of a system forcreating a differential blood count in FIG. 11) typically consists ofthe following core modules: detection of the cells, segmentation of thecells, and classification of the cells.

In particular in bone marrow smears, cells (typically white bloodcorpuscles—leukocytes) are mostly exist in the form of cell clusters(cell groups), i.e. the individual cells are directly adjacent to oneanother and are therefore difficult to segment, which complicates exactdifferentiation. Several methods addressing segmentation of leukocytesin bone marrow smears have been known from the literature. The majorityof methods are based on the watershed algorithm. The documents LEHMANN;T., W. OBERSCHELP; E. PELIKAN and R. REPGES: Bildverarbeitung für dieMedizin, Springer-Verlag, 1997, GONZALES, R. C., and R. E. WOODS:Digital Image Processing (3^(rd) Edition), Prentice-Hall, Inc., UpperSaddle River, N.J., USA, 2006, show an application of this watershedalgorithm in digital image processing. The most widely used methods ofsegmenting leukocytes in bone marrow smears will be mentioned below.

The documents NILSSON, B, and A. HEYDEN: Model-based Segmentation ofLeukocytes Clusters. In: ICPR '02: Proceedings of the 16^(th)International Conference on Pattern Recognition (ICPR, 02, Volume 1,page 10,727, Washington, D.C., USA, 2002. IEEE Computer Society, andNILSSON, B., and A. HEYDEN: Segmentation of complex cell clusters inmicroscopic images: application to bone marrow samples. Cytometry,66(1): 24-31, 2005, show a method of segmenting complex cell clusters inmicroscopic images. By means of this method, the cell clusters areinitially separated from the background by a threshold-value method. Inorder to separate leukocytes within a cluster, the backgroundsegmentation is subjected to weighted distance transformation, and theresult is subdivided into regions by means of the watershed algorithm.Since the watershed step results in over-segmentation, adjacent regionsare merged on the basis of such features as “roundness”, “surface area”etc. The result is the segmentation of cells.

The document PARK, J., and J. KELLER: Snakes on the Watershed. IEEETransactions on Pattern Analysis and Machine Intelligence, 23(10):1,201-1,205, 2001, shows a so-called snakes-on-the-watershed method. Asin the preceding method, distance transformation is applied tobackground segmentation. The output of the subsequent watershedalgorithm is used for initializing the “snake” zones of the “snake”algorithm. By means of the “snake” algorithm, the contour of eachleukocyte is then determined.

In the method of the document PARK, J.-S., and J. KELLER: Fuzzy patchlabel relaxation in bone marrow cell segmentation, in: Systems, Man, andCybernetics. 1997. ‘Computational Cybernetics and Simulation’, 1997 IEEEInternational Conference on, Volume 2, pages 1,133-1,138, Volume 2,October 1997, the watershed algorithm is applied directly to the inputimage. By means of stochastic methods, the resulting regions areassociated with the four classes of background, red blood corpuscles(erythrocytes), cytoplasm and cell nucleus.

In the method of the document JIANHUA, W., Z. LI, L. YANGBIN and Z.PINGPING: Image Segmentation Method based on Lifting Wavelet andWatershed Arithmetic, in: Electronic Measurement and Instruments, 2007.ICEMI '07. 8th International Conference on, pages 2-978-2-981, 16., 18.July 2007, 2007, images having lower resolutions are generated from theinput image by means of so-called wavelets (which developed when wavelettransformation was employed). Said images are then segmented with theaid of the watershed algorithm. The segmentations calculated at thedifferent stages of resolution are combined to obtain a high-qualitysegmentation result of the original image.

In the following, mention shall also be made of two further methods notbased on the above-mentioned watershed algorithm. The document HENGEN,H., S. L. SPOOR and M. C. PANDIT: Analysis of blood and bone marrowsmears using digital image processing techniques, in: M. SONKA & J. M.FITZPATRICK (Eds.): Society of Photo-Optical Instrumentation Engineers(SPIE) Conference Series, Volume 4.684 of the series Presented at theSociety of Photo-Optical Instrumentation Engineers (SPIE) Conference,pages 624-635, May 2002, shows a method wherein segmentation isperformed in that a distance transformation is calculated on the basisof a background estimation. By means of a threshold value, regions arethen generated on the distance image, said regions representing themidpoints of the cells, for example. Said regions are then used forinitializing a “region growing” algorithm so as to find the boundariesof the cells.

The document MONTSENY, E., P. SOBREVILLA and S. ROMANI: A fuzzy approachto white blood cells segmentation in color bone marrow images, in: FuzzySystems, 2004. Proceedings. 2004 IEEE International Conference on,Volume 1, pages 173-178, Volume 1, July 2004, shows a method whereineach pixel is associated with one of 53 color patterns. With the aid ofstochastic methods, each color pattern has one of three classes assignedto it, and thus classification of each pixel is achieved. The threeclasses are called region of interest, undefined region, and region notof interest.

All of the methods introduced herein have the disadvantage that reliablesegmentation of cells that are present in cell clusters is not effectedor is effected only insufficiently. In particular methods based on thewatershed algorithm tend to subdivide one cell into several individualregions, i.e. a cell is over-segmented.

SUMMARY

According to an embodiment, a method of segmenting biological cells in apicture so that the biological cells represent a foreground of thepicture may have the steps of applying a first fast marching algorithmto the picture or to a pre-processed version of same so as to achieve afirst fast marching image, the first fast marching algorithm startingfrom a background of the picture, and a velocity function of the firstfast marching algorithm being based on a first edge-strength image ofthe picture; segmenting the first fast marching image or afurther-processed version of same into a plurality of homogeneousregions; mapping each of the homogeneous regions to one node,respectively, of a graph so that nodes of adjacent homogeneous regionsare connected to one another and so that the graph has roots whichcorrespond to homogeneous regions located at cell centers; classifying,on the basis of the graph, each homogeneous region either as backgroundor as foreground; and applying a second fast marching algorithm withinthe homogeneous regions classified as foreground to a secondedge-strength image so as to segment the foreground into individualbiological cells, the second fast marching algorithm starting from thosehomogeneous regions which correspond to the roots of the graph.

According to another embodiment, an apparatus for segmenting biologicalcells in a picture so that the biological cells represent a foregroundof the picture may have: a fast marching processor configured to apply afirst fast marching algorithm to a picture or a pre-processed version ofsame so as to achieve a first fast marching image, the first fastmarching algorithm starting from a background of the picture, and avelocity function of the first fast marching algorithm being based on afirst edge-strength image of the picture; a segmenter configured tosegment the first fast marching image or a further-processed version ofsame into a plurality of homogeneous regions; a mapper configured to mapeach of the homogeneous regions to one node, respectively, of a graph sothat nodes of adjacent homogeneous regions are interconnected and thatthe graph has roots which correspond to homogeneous regions located atcell centers; and a classifier configured to classify each homogeneousregion either as background or as foreground on the basis of the graph;said fast marching processor being further configured to apply a secondfast marching algorithm within the homogeneous regions classified asforeground to a second edge-strength image so as to segment theforeground into, individual biological cells, said fast marchingprocessor being configured such that said second fast marching algorithmstarts from those homogeneous regions that correspond to the roots ofthe graph.

Another embodiment may have a computer program including a program codefor performing the method of segmenting biological cells in a picture,so that the biological cells represent a foreground of the picture,which method may have the steps of applying a first fast marchingalgorithm to the picture or to a pre-processed version of same so as toachieve a first fast marching image, the first fast marching algorithmstarting from a background of the picture, and a velocity function ofthe first fast marching algorithm being based on a first edge-strengthimage of the picture; segmenting the first fast marching image or afurther-processed version of same into a plurality of homogeneousregions; mapping each of the homogeneous regions to one node,respectively, of a graph so that nodes of adjacent homogeneous regionsare connected to one another and so that the graph has roots whichcorrespond to homogeneous regions located at cell centers; classifying,on the basis of the graph, each homogeneous region either as backgroundor as foreground; and applying a second fast marching algorithm withinthe homogeneous regions classified as foreground to a secondedge-strength image so as to segment the foreground into individualbiological cells, the second fast marching algorithm starting from thosehomogeneous regions which correspond to the roots of the graph, when theprogram runs on a computer.

Embodiments of the present invention provide a method of segmentingbiological cells in a picture so that the biological cells represent aforeground of the picture, comprising a step of applying a first fastmarching algorithm to the picture or to a pre-processed version of samein order to obtain a first fast marching image. The first fast marchingalgorithm starts from a background of the picture. A velocity functionof the first fast marching algorithm is based on an edge-strength imageof the picture.

In addition, the method comprises a step of segmenting the first fastmarching image or a further-processed version of same into a pluralityof homogeneous regions.

In this context, a homogeneous region may be contiguous in mathematicalterms and may therefore also be referred to as a contiguous homogeneousregion.

Furthermore, the method comprises a step of mapping each of thehomogeneous regions to one node of a graph, respectively. Said mappingis effected such that nodes of adjacent homogeneous regions areinterconnected and that the graph comprises roots which correspond tohomogeneous regions located at cell centers.

In addition, the method comprises a step of classifying each homogeneousregion either as background or foreground on the basis of the graph.

Moreover, the method comprises a step of applying a second fast marchingalgorithm. The second fast marching algorithm is applied, within thehomogeneous regions classified as foreground, to a second edge-strengthimage so as to segment the foreground into individual biological cells.The second fast marching algorithm starts from the homogeneous regionscorresponding to the roots of the graph.

It is a core idea of the present invention that improved segmentation ofcells in a picture may be achieved when, on the basis of anedge-strength image of the picture, a first fast marching image isproduced which is segmented into a plurality of homogeneous regions soas to subdivide said homogeneous regions into foreground and backgroundon the basis of a graph, and when a second fast marching algorithm isapplied within the homogeneous regions classified as foreground so as tosegment the foreground into individual biological cells.

One advantage of embodiments of the present invention is that thebiological cells present in the picture may be amplified by applying thefirst fast marching algorithm to obtain the first fast marching image.This enables easier segmentation of the first fast marching image or ofthe further-processed version of same into the plurality of homogeneousregions as compared to an image wherein the first fast marchingalgorithm was not applied.

A further advantage of embodiments of the present invention is thatsimple and unambiguous decision criteria may be formed by mapping thehomogeneous regions to nodes of a graph. Said decision criteria may beapplied in a simple manner in the step of classifying the homogeneousregions as background or foreground.

A further advantage of embodiments of the present invention is thatclassifying the homogeneous regions as background or foreground enablesthat the second fast marching algorithm can only be applied within thehomogeneous regions classified as foreground. Thus, both the computingexpenditure and the computing time involved are reduced for the secondfast marching algorithm.

Therefore, embodiments of the present invention provide a method whichenables improved segmentation of biological cells in a picture ascompared to the methods cited in the introductory part of thisapplication.

In accordance with some embodiments of the present invention, thevelocity function of the first fast marching algorithm may further bebased on a distance function indicating, for each pixel of the picture,a distance of a color value of the respective pixel from a cellestimation color value associated with the respective pixel. Forexample, said cell estimation color value may be an estimated foregroundcolor of that leukocyte which is closest with regard to the pixel.

By using this distance function as the second parameter of the velocityfunction of the first fast marching algorithm, color information of thepicture may also be used in addition to edge information (which is usedin the edge-strength image of the picture) in order to obtain animproved fast marching image. Utilization of the distance function andof the edge-strength image as parameters for the velocity functionenables, for example, that the velocity function is slow in an area ofan edge (for example an edge between the cell and the background), andis fast in an area of a large distance between the color value and thecell estimation color value (for example in the background). Thisenables, for example, that the first fast marching algorithm firstlyprocesses the background of the picture and lastly processes the cellnuclei (for example when the cell estimation color value is a colorvalue of a cell nucleus or at least of a cell). On the basis of thepoints in time at which the respective pixels of the picture areprocessed by the first fast marching algorithm, i.e. of the so-calledarrival time of the fast marching front, the first fast marching imagemay be formed, for example as a gray-level image.

In accordance with further embodiments of the present invention, thesecond fast marching algorithm may be configured such that a meetingline between a first fast marching front, which starts from a first cellcenter of a first biological cell, and a second fast marching front,which starts from a second cell center of a second biological cell,forms a boundary between the first biological cell and the secondbiological cell. By forming boundaries between different cells on thebasis of meeting lines of fast marching fronts of the second fastmarching algorithm, over-segmentation of cells, as it may be the case,for example, in the above-mentioned watershed algorithm, can be (almost)ruled out. In addition, the formation of boundaries between the cells onthe basis of meeting lines between the fast marching fronts enables atermination criterion for the fast marching algorithm, for example suchthat a fast marching front which meets another fast marching front willno longer propagate at this meeting line. This enables time-efficientimplementation of the second fast marching algorithm.

In accordance with some embodiments, the second edge-strength image usedin the step of applying the second fast marching algorithm may be thesame as the first edge-strength image, whereby computing expenditure ofthe method may be significantly reduced.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 shows a flowchart of a method in accordance with an embodiment ofthe present invention;

FIG. 2 shows a flowchart of a method in accordance with a furtherembodiment of the present invention;

FIG. 3 a shows an exemplary original image as may be used as an inputfor the method in accordance with FIG. 2;

FIG. 3 b shows the image of FIG. 3 a once a pre-processing step of themethod in accordance with FIG. 2 has been performed;

FIG. 4 shows a flowchart which depicts a partitioning of a second stepof the method of FIG. 2 into sub-steps;

FIGS. 5 a to 5 e show intermediate images as may be formed whenperforming the steps of the flowchart shown in FIG. 4;

FIG. 6 shows a flowchart which depicts a subdivision of a third step ofthe method of FIG. 2 into sub-steps;

FIGS. 7 a to 7 f show intermediate images as may be formed whenperforming the steps of the flowchart of FIG. 6;

FIG. 8 a shows an exemplary image comprising a plurality of homogeneousregions;

FIG. 8 b shows a graph wherein the regions of FIG. 8 a are mapped tonodes of the graph;

FIG. 9 a shows the input image of FIG. 3 a;

FIG. 9 b shows an output image of the method of the flowchart in FIG. 2while using the image of FIG. 9 a as the input image for the method;

FIG. 10 shows a block diagram of an apparatus in accordance with anembodiment of the present invention; and

FIG. 11 shows an image of a CAM system.

DETAILED DESCRIPTION OF THE INVENTION

Before embodiments of the present invention will be explained in moredetail below with reference to the accompanying figures, it shall benoted that elements that are identical or identical in function will bedesignated by the same reference numerals, and that repeated descriptionof said elements will be dispensed with. Therefore, descriptions ofelements having identical reference numerals are interchangeable.

FIG. 1 shows a flowchart of a method 100 of segmenting biological cellsin a picture, so that the biological cells represent a foreground of thepicture, in accordance with an embodiment of the present invention. Abiological cell may be a white blood corpuscle (leukocyte), a red bloodcorpuscle (erythrocyte) or any other biological cell. For example, thepicture may comprise a plurality of biological cells of different celltypes. The cells may be present in so-called cell clusters (cellgroups), for example. The picture may be an RGB (red/green/blue) image,a CMY (cyan/magenta/yellow) image, an HSI (hue/saturation/intensity)image, or a gray-level image. An RGB image may, for example, have adepth of 8 bits per channel and pixel (also referred to as an imagepoint, for example). The picture may be present on a digital storagedevice, for example, or may be taken in a separate step of the method.

The method 100 comprises a step 110 of applying a first fast marchingalgorithm to the picture or to a pre-processed version of same in orderto obtain a first fast marching image. The first fast marching algorithmstarts from a background of the picture. A velocity function of thefirst fast marching algorithm is based on an edge-strength image of thepicture. The edge-strength image of the picture may be created, e.g., ina previous step or may already be present on a storage medium and bemade available to the method.

In addition, the method 100 comprises a step 120 segmenting the firstfast marching image or a further-processed version of same into aplurality of homogeneous regions. The further-processed version of thefast marching image may be created, e.g., by smoothing the fast marchingimage in a further step of the method, for example by applying alow-pass filter to the first fast marching image. This enables filteringout artefacts that have developed in step 110. For example, ahomogeneous region may be characterized in that pixels of the fastmarching image which are associated with a common homogeneous regionexhibit similar color values in a predefined area. A color value of ahomogeneous region may be a mean value of color values of pixels of thehomogeneous region, for example.

In the following, a homogeneous region may also be referred to as aregion for short.

A color value of a pixel, or a sample of a homogeneous region, may alsobe referred to as a sample of the pixel or a sample of the homogeneousregion in the following. For example, a sample may describe a strengthof the individual color channels or of a gray channel of the pixel or ofthe homogeneous region, or a brightness of the pixel or of thehomogeneous region.

In the first fast marching image, a biological cell may be representedby a plurality of homogeneous regions. For example, a cell nucleus of abiological cell may be represented by one homogeneous region, and acytoplasm of the cell may be represented by one or more homogeneousregions. Color values of homogeneous regions representing a cell maydecrease from the inside (from the cell nucleus) toward the outside(toward the edge of the cytoplasm). Since color gradients of biologicalcells are typically similar, different biological cells of the same typemay typically have a similar number of homogeneous regions.

In addition, the method 100 comprises a step 130 of mapping each of thehomogeneous regions to one node of a graph, respectively, such thatnodes of adjacent homogeneous regions are interconnected and that thegraph comprises roots which correspond to homogeneous regions located atcell centers.

A cell center of a biological cell may also be referred to as a cellmidpoint and may be a cell nucleus of the biological cell, for example.A root of the graph may be defined, e.g., as a homogeneous region whichis not adjacent to any homogeneous region whose color value is higher(brighter) than its own. A leaf of the graph may be defined, e.g., as ahomogeneous region which is not adjacent to any homogeneous region whosecolor value is smaller (darker) than its own. In other words, roots ofthe graph may form local maxima, and leaves of the graph may form localminima, for example with regard to the color values of the homogeneousregions.

In addition, the method 100 comprises a step 140 of classifying, on thebasis of the graph, each homogeneous region either as background orforeground.

Classification of a homogeneous region as background or foreground maybe effected, for example, on the basis of a position of a node—to whichthe homogeneous region was mapped—within the graph. On the basis of theposition of the node within the graph in connection with a decisioncriterion, one may then determine for each homogeneous region whethersame is classified as foreground or background. Any homogeneous regionsclassified as background may be combined and may yield final backgroundsegmentation. The remainder of the homogeneous regions may be combinedas the foreground.

Moreover, the method 100 comprises a step 150 applying a second fastmarching algorithm. The second fast marching algorithm is applied,within the homogeneous regions classified as foreground, to a secondedge-strength image so as to segment the foreground into individualbiological cells. The second fast marching algorithm starts from thehomogeneous regions corresponding to the roots of the graph. In otherwords, the second fast marching algorithm starts from the homogeneousregions that are located at cell centers. In this context, the secondfast marching algorithm may start in parallel from any homogeneousregion located in a cell center, so that several fast marching frontswill simultaneously propagate within the homogeneous regions classifiedas foreground. Since the second fast marching algorithm is applied onlywithin the homogeneous regions classified as foreground, a velocityfunction of the second fast marching algorithm within the homogeneousregions classified as background may be zero. An output of the secondfast marching algorithm, i.e. a second fast marching image, for example,thus is also an output of the method 100. Thus, in this second fastmarching image, the foreground is separate from the background, and, inparticular, the biological cells are segmented, i.e. separated from oneanother. The mutually segmented cells may now be classified in a furtherstep.

FIG. 2 shows a flowchart of a method 200 of segmenting biological cellsin a picture, so that the biological cells represent a foreground of thepicture, in accordance with a further embodiment of the presentinvention. The method 200 represents a more detailed version of themethod 100 as was described above. The method 200 is described by way ofexample of segmentation of white blood cells (leukocytes) within bonemarrow smears. Even though the method is utilized in connection withsegmentation of leukocytes in bone marrow smears, this method 200—justlike the method 100, which is an abstract version of the method 200—mayalso be utilized for segmenting any other biological cells in pictures(e.g. images, photos or the like).

The flowchart in FIG. 2 shows the approximate course of the(segmentation) method 200.

An original image 201 (which may also be referred to as a picture), asis shown in FIG. 3 a for example, serves as an input for the method 200.A first step 202 of pre-processing the original image 201 of the method200 comprises pre-processing the original image 201, for example toeliminate disturbances in the original image 201. An output of thepre-processing step 202 is, e.g., a pre-processed version 203 (which wasalready mentioned above) of the original image 201. The pre-processedversion 203 of the original image 201 may be seen in FIG. 3 b.

A second step 204 of amplifying the cells of the method 200 comprisesprocessing the pre-processed version 203 of the original image 201 suchthat the cells present in the original image 201 become more prominent.An output of the step 204 is a first fast marching image 205 (which wasalready mentioned above), for example.

A third step 206 of detecting the cell midpoints and of segmenting thebackground comprises segmenting the first fast marching image 205 intocell midpoints, foreground and background. An output of the third step206 may be a graph 207 (which was already mentioned above) of nodes.

A fourth step 208 of separating the cells of the method 200 comprisesseparating—on the basis of the graph 207—the individual biological cellsthat are present in the original image 201. An output of the fourth step208 is an output image 209 (which was already mentioned above), which isalso an output image of the method 200 and wherein the individualbiological cells are segmented, i.e. separated from one another. Theoutput image 209 may be a list of foreground regions, for example, eachforeground region representing precisely one cell.

The individual steps of the method 200 shall be described in detailbelow. It is to be noted that the second step 204 of the method 200comprises several sub-steps shown in a flowchart in FIG. 4. In addition,it shall be noted that the third step 206 of the method 200 comprisesseveral sub-steps shown in a flowchart in FIG. 6.

The first step 202 of pre-processing the method 200 serves to eliminatedisturbances in the original image 201 (which may also be referred to asan input image or a picture). Disturbances may be, e.g., undesired colorgradients in the background of the original image 201 which havedeveloped due to non-uniform illumination or due to remainders of burstcells. FIG. 3 a shows how the original image 201 serves as the basis forthe method 200 of segmenting the biological cells in the original image201. FIG. 3 b shows the further-processed version 203 of the originalimage 201 following the pre-processing step 202. In FIGS. 3 a and 3 b, abackground 301 of the original image 201 and of the further-processedversion 203 of same is depicted to be bright, whereas cell components302 are represented by dark areas. Comparing the original image 201 andthe further-processed version 203 of same reveals that color gradientswere calculated out from the background 301. An exemplary color gradient303 in the original image 201 is hatched. Said color gradient 303 nolonger exists in the further-processed version 203 of the original image201 because of the pre-processing step 202.

The pre-processing step 202 is optional, i.e. in accordance with someembodiments the pre-processing step 202 may also be omitted, so that thestep 204 of amplifying the cells onto the original image 201 isperformed.

The step 204 of amplifying the cells comprises processing the originalimage 201 or the further-processed version 203 of same (which was formedby the pre-processing step 202) such that the cells (which are presentin the original image 201) become more prominent. In this manner,segmentation of the leukocytes (for example in the step 206 of detectingthe cell midpoints and of segmenting the background) may be achievedmore easily later on. To gain a deeper understanding of the step 204 ofamplifying the cells, the course of this step is shown in the flowchartin FIG. 4. In FIGS. 5 a-5 e, the individual sub-steps shown in theflowchart of FIG. 4 are illustrated by means of images.

It shall be assumed in the following that the pre-processed version 203of the original image 201 is used as the input for the step 204.

A sub-step 401 of determining pixels that are very likely to belong toleukocytes, i.e. to the foreground of the original image 201, comprisesdetermining the pixels of the pre-processed version 203 of the originalimage 201 which certainly belong to leukocytes, i.e. to the foregroundof the original image 201.

In accordance with some embodiments, said pixels which are very likelyto belong to leukocytes may be determined, for example, on the basis ofa simple color threshold-value method. For example, a pixel may bedetermined as being very likely to belong to a leukocyte when a colorvalue of said pixel exceeds a specific limiting value. A color value ofa pixel may be based on red, green, blue values of the pixel (for an RGBimage, for example). In their cell centers (i.e. in their cell nuclei)cells typically have color distributions that are known, so that a pixelthat is a component of the cell nucleus may be easily determined on thebasis of its color value.

In accordance with some further embodiments, the further-processedversion 203 of the original image 201 may be transformed into agray-level image shown in FIG. 5 b. Transformation of thefurther-processed version 203 of the original image 201 into thegray-level image shown in FIG. 5 b may be performed, e.g., on the basisof the following formula

$\begin{matrix}{{I_{gbatan}( {x,y} )} = {{arc}\;{{\tan( \frac{I_{green}( {x,y} )}{I_{blue}( {x,y} )} )}.}}} & (1)\end{matrix}$I_(green) and I_(blue) are the green and blue channels of the originalimage 201 and of the further-processed version 203 of same,respectively. (x,y) indicate the coordinates of a pixel within theoriginal image 201 or within the further-processed version 203 of same.

The original image 201 or the further-processed version 203 of theoriginal image 201 may be transformed into the gray-level image suchthat, for example, a likelihood (of a pixel) of belonging to theforeground continuously increases or decreases along a gray-level scaleof the gray-level image.

The above formula (1) reveals that a gray level I_(gbatan)(x,y) isdetermined for each pixel of the further-processed version 203 of theoriginal image 201. To obtain background and/or foreground segmentation,a first threshold value may be determined on this gray-level image. Saidfirst threshold value may be determined, for example, using theso-called Otsu method. An output of this threshold-value method are thepixels that are very likely to belong to the foreground of the originalimage 201 and/or of the further-processed version 203 of same and thusare very likely to belong to leukocytes. Pixels that are very likely tonot belong to leukocytes may be pixels belonging to the background orbelonging to artefacts. Consequently, a second threshold value may beformed on the basis of the pixels not are very likely to belong to theforeground (i.e. on the basis of all of the pixels except for the pixelsthat are very likely to belong to the foreground), so as to determineany pixels that are very likely to be located in the background (belongto the background). This may be performed, for example, in a sub-step402 (of the step 204 of the method 200) of determining pixels that arevery likely to be located in the background. Determination of the pixelsthat are that are very likely to be located in the background may beeffected, for example, at a second stage of the Otsu method. In otherwords, the second threshold value may be determined by invoking the Otsumethod once again; those pixels that are very likely to belong to theforeground of the original image 201 or of the further-processed version203 of same have been removed from the calculation for the secondthreshold value. Even though in the flowchart shown in FIG. 4 the twosub-steps 401, 402 are not depicted to be directly mutually successive,the two sub-steps 401, 402 may be performed directly one after the otheror even in a common step, in accordance with further embodiments.

In other words, the method 200 may comprise a step of determining pixelswhich are very likely to belong to a background of the picture (of theoriginal image 201) and of determining pixels which are very likely tobelong to a foreground of the picture (of the original image 201) whileusing a (two-stage) threshold-value method.

In addition, in accordance with embodiments of the present invention,the step of determining pixels which are very likely to belong to abackground of the picture and of determining pixels which are verylikely to belong to a foreground of the picture may comprisetransforming the picture (the original image 201) or thefurther-processed version 203 of same into a gray-level image (describedabove) to determine, on the basis of a first threshold value, thosepixels that are very likely to belong to the foreground, and tosubsequently determine, on the basis of a second threshold value, from aquantity of pixels that are very likely to not belong to the foreground,those pixels that are very likely to belong to the background.

FIG. 5 c shows the gray-level image of FIG. 5 b following thedetermination of those pixels that are very likely to belong toleukocytes, and following the determination of those pixels that arevery likely to be located in the background. The image in FIG. 5 c thusshows coarse segmentation of foreground (depicted in black) andbackground (depicted in white).

On the basis of this segmentation, the foreground color of the closestleukocyte (with regard to the pixel) may be estimated for each pixel ofthe image (of the original image 201 or of the further-processed version203 of same) in a sub-step 403 of estimating the foreground color foreach pixel and/or of associating a cell estimation color value with eachpixel (of the step 204 of the method 200) on the basis of thissegmentation. The foreground color is estimated on the basis of thosepixels which are very likely to belong to leukocytes and were determinedby the above-mentioned threshold-value method. A foreground color of aleukocyte may also be referred to as a cell estimation color value or asa cell estimation sample.

A first cell estimation color value associated with a first pixel may bedetermined on the basis of a predefined minimum number of color valuesof pixels which are very likely to belong to a foreground of the pictureand are located in an area around the first pixel. This is effected foreach pixel, so that each pixel of the picture (of the original image201) has precisely one cell estimation color value associated with it.

In other words, the foreground color of the closest leukocyte may beestimated, for each pixel of the image, in that in a window around eachpixel (image point) the mean value (of the color values of the pixels inthe window which are very likely to belong to the foreground) iscalculated and the result is assigned to the pixel in the center (of thewindow) as the color estimation value. If the number of foregroundpixels (image points that are very likely to belong to the foreground)that are available in the window falls below a specific threshold (aminimum number), the window size may be increased as long as sufficientpixels (image points that are very likely to belong to the foreground)are available. In this context, for example, a window may have a square,rectangular, round, hexagonal or any other shape. In other words, a cellestimation color value of a pixel may correspond to a central tendencyof color values of pixels in a predetermined environment of therespective pixel which are very likely to belong to the foreground.

In accordance with further embodiments, the sub-step 403 of estimatingthe foreground color for each pixel may also be performed such that eachpixel which is not very likely to belong to the foreground of theoriginal image 201 has a cell estimation color value associated with it.In other words, it is not absolutely necessary that each pixel of theoriginal image 201 has a new cell estimation color value associated withit. For example, a cell estimation color value of a pixel that is verylikely to belong to the foreground and is thus located within abiological cell may be identical with its color value, since the pixelis obviously already part of the closest leukocyte.

FIG. 5 d shows a result of the sub-step 403 of estimating the foregroundcolor for each pixel.

The step 204 of the method 200 further comprises a sub-step 404 ofcalculating an edge-strength image. Even though in the flowchart in FIG.4 calculation of the edge-strength image is performed after the sub-step403 of estimating the foreground color for each pixel, calculation ofthe edge-strength image may also be performed prior to said sub-step 403and prior to the sub-step 401 of determining pixels that are very likelyto belong to leukocytes, since the edge-strength image is created on thebasis of the original image 201 or of the further processed version 203of same. The edge-strength image may be created, for example, byhigh-pass filtering the original image 201 or the further-processedversion 203 of same, so that only high frequencies present in theoriginal image 201 or in the further-processed version 203 of same canpass these high-pass filters, and lower frequencies are filtered out.High frequencies typically occur at transitions with major colorchanges, such as is the case, for example, between the foreground andthe background and/or between the cell and the background. Locationshaving high frequencies in the original image 201 or in thefurther-processed version 203 of same thus represent edges between cellsand background. An output of the sub-step 404 is the edge-strength imagerepresented in FIG. 5 a. Edges are represented by bright areas here.Areas having no edge are shown to be dark. As may be seen from theedge-strength image in FIG. 5 a, the sub-step 404 of determining and/orcalculating the edge-strength image comprises assigning, to each pixelof the picture (of the original image 201) or of the further-processedversion 203 of same, an edge-strength value which depends on a (color)gradient of a color curve of the original image 201 or of thefurther-processed version 203 of same at the respective pixel.

In the specific embodiment shown here, the edge-strength image may becalculated using an algorithm of Di Zenzo, as is shown in the documentDI ZENZO, S.: A note on the gradient of a multi-image. Comput. VisionGraph. Image Process, 33(1): 116-125, 1986.

In addition, the step 204 of the method 200 comprises a sub-step 405 ofapplying a first fast marching algorithm. The sub-step 405 of applyingthe first fast marching algorithm may be the previously mentioned step110 of the method 100, for example. As was already mentioned above, thestep 204 of the method 200 serves to amplify the cells, so that theybecome more prominent. The first fast marching algorithm is used toachieve precisely this amplification. It is initialized with a quantityof pixels (image points) that are very likely to be located in thebackground. Said pixels or image points were determined in the sub-step402 of the step 204 of the method 200. On the basis of color informationand edge strengths, the propagation of a fast marching front of thefirst fast marching algorithm is influenced such that it initiallypasses through the entire background, then the foreground, and, lastly,the cell centers.

The previously mentioned color information is based on the cellestimation color values for each pixel which were determined in thesub-step 403. The edge strengths are based on the edge-strength imagedetermined in the sub-step 404. The fast marching front of the firstfast marching algorithm reaches pixels of the cells (the pixels whichare very likely to belong to leukocytes and were determined in step 401)much later than those pixels that are very likely to be located in thebackground. This results in a high arrival time within the cells and ina low arrival time in the background. If said arrival times areinterpreted as gray levels in an image (in the first fast marching image205 as is depicted in FIG. 5 e), the cells are very prominent and willthen be easier to detect and to segment in the next step (in step 206 ofthe method 200).

In addition to requiring the initialization points (pixels that are verylikely to be located in the background) from which the first fastmarching algorithm starts and which may be calculated, as was described,by a two-stage OTSU method, for example, the first fast marchingalgorithm necessitates also a velocity function. The velocity functionused in the specific embodiment shown here is indicated in the followingformula (2):

$\begin{matrix}{{F_{bg}( {x,y} )} = {\frac{{D( {x,y} )}^{\alpha} + \theta}{\gamma + {I_{grad}( {x,y} )}}.}} & (2)\end{matrix}$

F_(bg)(x,y) indicates the velocity of the fast marching front of thefirst fast marching algorithm in a pixel having the coordinates (x, y).D(x,y) indicates the distance of the color value of the current pixelhaving the coordinates (x, y) from an estimation of the foreground color(of the cell estimation color value calculated in the sub-step 403 foreach pixel). I_(grad) is i the edge-strength image calculated in thespecific embodiment with the aid of the algorithm of Di Zenzo from theoriginal image 201 or the further-processed version 203 of same.I_(grad)(x,y) indicates an edge-strength value of the current pixel. αand θ are parameters that influence how both components (the distanceD(x,y) and the edge-strength value I_(grad)(x,y)) of the formula areweighted. γ is an additional parameter which prevents that thedenominator of the velocity function becomes zero, and in the specificembodiment may have the value of 0.00001.

As was already mentioned above, FIG. 5 e shows the output of the firstfast marching algorithm in the form of the first fast marching image205. Dark areas characterize areas having a low arrival time of the fastmarching front of the first fast marching algorithm, and bright areascharacterize areas having a high arrival time of the fast marching frontof the first fast marching algorithm. It becomes clear that cell nuclei501 are represented to be brightest, since they have a highest arrivaltime. This becomes apparent from formula (2) for the velocity functionof the first fast marching algorithm, since a distance D(x,y) of a pixellocated in a cell nucleus 501 from its cell estimation color value(estimated foreground color of the closest leukocyte) is typically smalland may even become zero, whereby the numerator of the formula (2) forthe velocity function of the first fast marching algorithm is determinedonly by means of θ, and whereby the velocity function in this area isthus slower than in areas wherein a distance D(x,y) between a colorvalue of pixel and its associated cell estimation color value is large.

In addition, pixels located in cell nuclei 501 are separated by an edgefrom the background, i.e. from the initialization points, of the firstfast marching algorithm. Therefore, to reach a pixel located in a cellnucleus 501 of a leukocytes, the fast marching front may first get pastan edge between the foreground and the background. However, this fastmarching front moves slowly in the area of the edges due to theedge-strength image in the denominator of the formula (2) for thevelocity function. As a result, the time of arrival of the fast marchingfront at the pixels located in cell nuclei 501 increases even more. Byapplying the first fast marching algorithm on the basis of this colorinformation and edge strengths, one achieves that the individual cellsare more prominent than is the case in the original image 201 (as isshown in FIG. 3 a) or in the further-processed version 203 of same (asis shown in FIG. 3 b).

The first fast marching image 205 serves as an input for the third step206 of the method 200 of detecting the cell midpoints (cell centers thatmay be located in cell nuclei 501) and of segmenting the background.

In the specific embodiment of the method 200, detection and segmentationof leukocytes is performed in several steps, which are depicted assub-steps of the step 206 in a flowchart in FIG. 6.

A sub-step 601 of Gaussian smoothing of the step 206 of the method 200comprises—in order to balance out any artefacts resulting from theprevious step (the step 204)—heavily smoothing the output of the firstfast marching algorithm (the first fast marching image 205). From aspectral point of view, Gaussian smoothing of the first fast marchingimage 205 is low-pass filtering of the first fast marching image 205. Inaccordance with further embodiments, the first fast marching image 205may also be smoothened using a different low-pass function.

An output of the step 601 of Gaussian smoothing is depicted as asmoothened fast marching output 602 in FIG. 7 a. FIG. 7 b shows amagnification of a cell 703 from the smoothened fast marching output602. One may see from FIGS. 7 a and 7 b that cell nuclei of theleukocytes are represented by bright color regions, and that thebrightness of the leukocytes decreases toward the edge (toward thecytoplasm). The background is depicted as a dark area in FIGS. 7 a and 7b.

A further sub-step 603 of segmenting the smoothened fast marching output602 into a plurality of homogeneous regions (of the step 206 of themethod 200) comprises segmenting the smoothened fast marching output 602into homogeneous regions. In the specific embodiment of the method 200,the sub-step 603 comprises applying the so-called color structure codealgorithm, which is shown in the document PRIESE, L. and P. STURM:Introduction to the Color Structure Code and it Implementation, 2003, tothe smoothened fast marching output 602. Said color structure codealgorithm serves to identify the homogeneous regions. The sub-step 603(of the step 206 of the method 200) may be, e.g., the step 120 ofsegmenting the first fast marching image or the further-processedversion of same into a plurality of homogeneous regions. Thefurther-processed version of the first fast marching image is thesmoothened fast marching output 602. As the output of the sub-step 603,one will obtain a list 604 of homogeneous regions found.

Since the background of the first fast marching image 205 and of thesmoothened fast marching output 602 is a dark area with homogeneouscoloration, the major part of the background may be represented by asingle homogeneous region. The cells (the leukocytes) themselves may berepresented by many small regions in each case, which are arranged likeconcentric circles (starting from a cell midpoint). Since biologicalcells are typically not present in an optimally circular shape, thehomogeneous regions may also be present in other concentric shapes,homogeneous regions having higher (brighter) color values (e.g. cellmidpoints) typically being surrounded by homogeneous regions havinglower (darker) color values (e.g. homogeneous regions representingcytoplasm).

The arrangement of the homogeneous regions may be seen in FIGS. 7 c and7 d as an output of the color structure code. FIG. 7 d shows a close-upof the cell 703 of FIG. 7 b. It becomes clear that the cell 703 has beensegmented into a plurality of homogeneous regions extending outward froma cell nucleus 704 of the cell 703. Typically, the cell nucleus 704 ofthe cell 703 is represented by a smaller number of homogeneous regionsthan a cytoplasm 705 of the cell 703. In addition, it becomes clear thata homogeneous region which is located in the cell nucleus 704 of thecell 703 has a brighter (a higher) color value than a homogeneous regionlocated in the cytoplasm 705 of the cell 703. A homogeneous regionlocated in the background typically has an even lower (darker) colorvalue than do homogeneous regions located in cytoplasms of cells.

The color structure code method serves to identify homogeneous regionsin an image. In this context, the pixels of the input image are supposedto be arranged in a hexagonal grid structure. Since the pixels of theinput image are normally arranged in a Cartesian manner, a mapping fromthe Cartesian grid to the hexagonal grid is defined. This hexagonal gridstructure is then recursively subdivided into small groups, which willbe called islands from now on. At the lowest level, seven pixels arecombined into one island of the level 0 in each case. This includes thepixel located at the center of the hexagon and its seven directlyadjacent pixels. All islands of the level 0, in turn, form the samehexagonal structure and may be combined into islands of the level 1 justlike at the lowest level. This is repeated recursively until one gets tothe topmost level, which consists of only one island.

The color structure code method consists of three phases. Theinitialization phase, the linking phase and the dividing phase. Theinitialization phase comprises verifying the pixels of each island ofthe level 0 as to their similarity, and grouping pixels that havesufficient similarity into so-called code elements. The linking phasecomprises generating the code elements of the next island up in that thecode elements of the level underneath are combined if they aresufficiently similar. This is repeated for all hierarchy levels,resulting in several trees of code elements. The dividing step isperformed whenever two adjacent code elements are not to be combined dueto their dissimilarity, but are already connected to one another viacommon code elements of lower levels. Subsequently, the code elementsunderneath are distributed to the two code elements. Once the method hasarrived at the highest level, several trees consisting of code elementsof different hierarchy levels will have been generated. Each tree nowrepresents a homogeneous region.

The similarity measure used is a threshold value that is compared to thedifference of two intensity values. If the difference is smaller, bothcode elements are considered to be similar.

In order to correctly select the free parameter of the color structurecode, a number of homogeneous regions that are to be achieved using thisalgorithm is specified in advance. With the aid of the bisection methodand of several color structure code runs, this free parameter is thendetermined such that the desired number of homogeneous regions is atleast approximated.

In accordance with some embodiments, pixels of the first fast marchingimage 205 or of a further-processed version of same (of the smoothenedfast marching output 602) are therefore combined into a commonhomogeneous region on the basis of color values of the pixels and of aneighborhood relationship of the pixels. In other words, segmenting 603of the first fast marching image 205 or of a further-processed versionof same (of the smoothened fast marching output 602) comprisescombining, in a hierarchical manner, pixel clusters of pixels of thefirst fast marching image 205 or of the further-processed version ofsame which are mutually adjacent and have a similar color value.

Observing the neighborhood relationship of the pixels is relevant sothat mutually spaced-apart cell components of different cells are notcombined into a common homogeneous region.

A color value of a homogeneous region may be a mean color value of allof the pixels of the homogeneous region, for example.

The step 206 of the method 200 further comprises a sub-step 605. Thesub-step 605 of mapping each of the homogeneous regions to one node,respectively, of a graph so that nodes of adjacent homogeneous regionsare connected to one another and so that the graph has roots whichcorrespond to homogeneous regions located at cell centers, has, as theinput, the list 604 of homogeneous regions as was determined by thesub-step 603 (the step 206 of the method 200). The sub-step 605 may bethe step 130 of the method 100, for example. An output of the sub-step605 is an ordered graph 606 of homogeneous regions. Each node of thegraph 606 represents one homogeneous region. Two nodes are connected toeach other when the associated homogeneous regions are adjacent. Themean value (the mean color value) of each homogeneous region indicatesthe order of the homogeneous region within the graph 606. That is,brighter homogeneous regions (e.g. regions located in cell midpoints orcell centers) are higher up in the graph and have a higher order, darkerhomogeneous regions (e.g. homogeneous regions located in the background)are located further down and have a lower order.

The association of homogeneous regions of cells with the graph 606 shallnow be explained by way of example by means of FIGS. 8 a and 8 b. FIG. 8a shows a picture (e.g. a sub-area of the image shown in FIG. 7 c)comprising several cells and several homogeneous regions. A density of ahatching of a homogeneous region represents the color value of therespective homogeneous region, a high color value representing a brighthomogeneous region and a low color value representing a dark homogeneousregion (e.g. background).

A first cell 801 in the picture 800 comprises a first homogeneous region802 located at a cell center (e.g. in a cell nucleus) of the first cell801. The first homogeneous region 802 is surrounded by a secondhomogeneous region 803, which may still belong to the cell nucleus ofthe first cell, but may also already be part of the cytoplasm of thefirst cell 801. A third homogeneous region 804 surrounds the secondhomogeneous region 803. A fourth region 805 surrounds the third region804. It becomes clear that the density of the hatchings of thehomogeneous regions 802-805 decreases in the outward direction withregard to the first cell 801, i.e. the color values of the regions802-805 decrease in the outward direction.

By analogy with the first cell 801, the same applies to a second cell806, which comprises a fifth homogeneous region 807 located at a cellcenter (e.g. in a cell nucleus) of the second cell 806. The fifthhomogeneous region 807 is surrounded by a sixth homogeneous region 808,which may still be part of the cell nucleus of the second cell 806, butmay also be part of a cytoplasm of the second cell 806. The sixthhomogeneous region 808 is surrounded by a seventh homogeneous region809, and said homogeneous region 809 is surrounded by an eighthhomogeneous region 810. For the second cell 806, too, the density of thehatchings of the homogeneous regions 807-810 decreases in the outwarddirection with regard to the second cell 806. A ninth homogeneous region811 has an even lower color value than the homogeneous regions 805 and810. A tenth homogeneous region 812 has a lowest color value of thehomogeneous regions of the picture 800. Therefore, the tenth homogeneousregion 812 may be referred to as the background of the picture 800.

FIG. 8 b shows the partitioning of the homogeneous regions of FIG. 8 awithin a graph, wherein each homogeneous region is mapped to preciselyone node, and wherein nodes of adjacent homogeneous regions areconnected to one other. It becomes clear that the first region 802,which is located at the cell center of the first cell 801, and the fifthhomogeneous region 807 located at the cell center of the second cell 806are roots of the graph in FIG. 8 b. In addition, the ninth homogeneousregion 811 is a root of the graph. A leaf of the graph is the tenthhomogeneous region 812. An arrangement of the regions within the graphmay be effected, as was already mentioned above, on the basis of themean color values of the homogeneous regions. This becomes clear, in thegraph in FIG. 8 b, in that even though forming a root of the graph, theninth homogeneous region 811 is arranged lower down in the order of thegraph than are the homogeneous regions 802 and 807.

Even though in the example shown here several homogeneous regions havethe same order within the graph and thus have the same mean color value,in further embodiments each homogeneous region may have a mean colorvalue of its own, and the graph may thus have an order which correspondsto the number of homogeneous regions.

In a further sub-step 607 (of the step 206 of the method 200), thehomogeneous regions obtained may now be classified. In this step 607 ofclassifying the regions, a decision is made as to which homogeneousregions make up the cell midpoints and, in addition, a decision is madeas to which regions belong to the background and to the foreground,respectively. The sub-step 607 of the step 206 of the method 200 may bethe step 140 of the method 100, for example. To obtain features for theclassification of the homogeneous regions, the ordered graph 606 wasbuilt from homogeneous regions in the sub-step 605.

Relevant features for the classification may be the following:

-   -   f₁: In an ascending manner, the shortest path to the closest        root.    -   f₂: In a descending manner, the shortest path to the closest        leaf.    -   f₃: In an ascending manner, the longest path to the closest        root.    -   f₄: In a descending manner, the longest path to the closest        leaf.    -   f₅: In an ascending manner, the longest path from any leaf to        any root through the current node.    -   f₆: Number of reachable roots by means of ascent.    -   f₇: Color value of the current region compared, in terms of        percentage, to the maximum and minimum color values in the        image.

In this application, a length of a path is determined to be the numberof edges of the graph 606 that are located between a starting node ofthe path and a terminal node of the path. One edge interconnects exactlytwo nodes. A length of a path thus is also the number of intermediatenodes located along the path from the starting node to the terminalnode, plus one.

The classification into foreground and background may be determined bymeans of several features. If one of said features is satisfied, thisregion will be background. In this context, Thr_(x) is a fixed thresholdvalue of the corresponding feature in each case. Features classifying aregion as background shall be cited below.

A first feature is f₇<Thr₁, which means that a color value of thecurrent region is smaller than a first predefined value Thr₁, and thatthis region consequently is classified as background, since regionshaving very low color values are background. With regard to the exampleof FIG. 8 b, for example, the homogeneous region 811 might be classifiedas background due to its low color value.

A second criterion is f₂=0, which means that a homogeneous regionbelonging to this node is a leaf of the graph. The leaves of the graphare background since leaves are local minima, and consequently, thehomogeneous region will also be classified as background.

A third criterion is f₅<Thr₂, which means that if a length of a longestpath, in an ascending direction with regard to the order of the graph606, from a leaf of the graph 606 to a root of the graph 606 through anode to which the homogeneous region is mapped is smaller than a secondpredefined value Thr₂, this homogeneous region will be classified asbackground. Or, in other words, if the longest path from a leaf to aroot through this region is too short, the region will be classified asbackground. In this manner, regions that have developed on account ofartefacts, such as, e.g., dirt in the picture (in the original image201), will be filtered out. This third criterion also filters out anyregions that are mapped to roots of the graph 606, wherein the path to aleaf is too short and which therefore cannot be a cell center.

A fourth criterion is f₇<Thr₃ and f₂<Thr₄ and f₆>1, which means that ifa color value of the respective homogeneous region is smaller than athird predefined value Thr₃, a length of a shortest path, in a directionthat is descending in terms of the order of the graph 606, from the nodeto which the homogeneous region is mapped to a closest leaf (with regardto the current node) of the graph 606 is smaller than a fourthpredefined value Thr₄, and if more than one root of the graph 606 may bereached, starting from the current node, in the ascending direction,said homogeneous region will be classified as background. In otherwords, this fourth criterion classifies as background such regions ofwhich several roots may be reached and which have a relatively low colorvalue. In the specific embodiment presented here, the fourth predefinedvalue Thr₄ may have the value 3. This 4^(th) criterion is useful forseparating cells from one another, when burst cells or erythrocytes (redblood corpuscles) exist between other cells and thus render separationdifficult.

Any regions that were classified as background are combined and yieldthe final background segmentation. The remaining regions are foreground.

A region is the center of a leukocyte (or of a cell) if it is a root ofthe graph 606 (met by the criterion f₁=0) and if it was classified asforeground rather than as background on the basis of the above-mentionedclassification (for example if the third criterion is not met).

Thus, an output of the step 207 are cell midpoints or cell centers(which are roots of the graph and are classified as foreground) and asegmentation into foreground and background. FIG. 7 e shows the cellmidpoints (depicted to be white), and FIG. 7 f shows the segmentationinto foreground and background, the foreground being depicted to bewhite and the background depicted to be black.

The output 207 of the sub-step 607 of the step 206 of the method 200and, thus, the output 207 of the step 206 of the method 200 serve as thefoundation for the fourth step 208 of the method 200 of separating theindividual cells (as is shown in the flowchart in FIG. 2). The fourthstep 208 of the method 200 may be the step 150 of the method 100, forexample.

This step 208 of the method 200 comprises separation on the basis of thepreviously calculated cell midpoints (or cell centers) and thesegmentation into background and foreground (i.e. on the basis of theoutput 207 of the step 206 of the method 200). This separation of thecells is realized with the aid of a second fast marching algorithm. Thecell midpoints calculated (shown in FIG. 7 e) are used as the initialfront of the second fast marching algorithm. In other words, the secondfast marching algorithm starts from the homogeneous regions located atcell centers which correspond to the roots of the graph 606.

A velocity function of the second fast marching algorithm has thefollowing form:

$\begin{matrix}{{F( {x,y} )} = \{ \begin{matrix}{0,} & {{{for}\mspace{14mu}( {x,y} )} \notin {ROI}_{fg}} \\{\frac{1}{ɛ + {I_{grad}( {x,y} )}},} & {{{for}\mspace{14mu}( {x,y} )} \in {{ROI}_{fg}.}}\end{matrix} } & (3)\end{matrix}$

ROI_(fg) is the foreground segmentation from the previous step (fromstep 206). Just like in the step 204, I_(grad) is an edge-strength imageand may be, e.g., the same edge-strength image as the edge-strengthimage from step 204. Just like in the velocity function of the firstfast marching algorithm, ε is a parameter which prevents the denominatorof the velocity function from becoming zero. One may see from theformula (3) for the velocity function of the second fast marchingalgorithm that the velocity function of the second fast marchingalgorithm adopts the value zero for homogeneous regions classified asbackground; i.e. not only does the second fast marching algorithm startin the foreground, but the fronts of the second fast marching algorithmend at the boundaries between foreground and background.

The second fast marching algorithm may have as many fronts as there arecell midpoints (cell centers), i.e. a separate fast marching frontstarts from each cell midpoint. By modifying this second fast marchingalgorithm, one may achieve that—if two fronts starting from differentcell midpoints or cell centers meet—said meeting points will be used asseparating lines between the cells, whereby unambiguous segmentation ofthe cells is enabled. In other words, a meeting line between a firstfast marching front, which starts from a first cell center of a firstbiological cell, and a second fast marching front, which starts from asecond cell center of a second biological cell, forms a boundary betweenthe first biological cell and the second biological cell.

The second fast marching image 209 generated by the second fast marchingalgorithm may be a final output image of the method 200 and provide alist of foreground regions, (at least approximately) each foregroundregion describing precisely one cell.

FIG. 9 a once again shows the original image 201 (the input image or thepicture) of the method 200, and FIG. 9 b shows the final output image(the second fast marching image 209) of the method 200. In the image inFIG. 9 b, one may clearly see that the individual cells have beensegmented from one another (divided from one another). In particular,one may see that no (or only minor) over-segmentation of the cells hasformed.

FIG. 10 shows a block diagram of an apparatus 1000 for segmentingbiological cells in a picture, so that the biological cells represent aforeground of the picture. The apparatus 1000 further comprises a fastmarching processor 1001, a segmenter 1002, a mapper 1003, and aclassifier 1004. The fast marching processor 1001 is configured to applya first fast marching algorithm to a picture 1005 or a pre-processedversion of same so as to obtain a first fast marching image 1006, thefirst fast marching algorithm starting from a background of the picture1005, and a velocity function of the first fast marching algorithm beingbased on a first edge-strength image 1007 of the picture 1005.

The segmenter 1002 is configured to segment the first fast marchingimage 1006 or a further-processed version of same into a plurality 1008of homogeneous regions.

The mapper 1003 is configured to map each of the homogeneous regions toone node, respectively, of a graph 1009 so that nodes of adjacenthomogeneous regions are interconnected and that the graph 1009 comprisesroots which correspond to homogeneous regions located at cell centers.

The classifier 1004 is configured to classify each homogeneous regioneither as background or as foreground on the basis of the graph 1009.One output of the classifier 1004 may be foreground and backgroundsegmentation 1010, for example.

The fast marching processor 1001 is further configured to apply a secondfast marching algorithm within the homogeneous regions classified asforeground to a second edge-strength image 1011 so as to segment theforeground into individual biological cells 1012, said second fastmarching algorithm starting from those homogeneous regions thatcorrespond to the roots of the graph.

In accordance with some embodiments, the fast marching processor 1001may also comprise two separate computing units, a first computing unitof the fast marching processor 1001 applying the first fast marchingalgorithm, and a second computing unit of the fast marching processor1001 applying the second fast marching algorithm.

The apparatus 1000, the fast marching processor 1001, the segmenter1002, the mapper 1003, and the classifier 1004 may be independenthardware units or parts of a processor of a computer or microcontrolleror digital signal processor, or may be realized as computer programs orcomputer program products for execution on a computer or microcontrolleror digital signal processor.

As was already mentioned above, a typical CAM system (shown in FIG. 11)consists of the core modules of detecting the cells, segmenting thecells, and classifying the cells. Embodiments of the present inventionaddress the first two modules of such a CAM system, detection andsegmentation of the cells. A method (the method 200) was presented, byway of example of bone marrow smears, said method addressing theproblem, in particular, that the white blood cells are present in cellclusters and that exact differentiation is rendered difficult as aresult. Since for implementing an automated approach (based on a CAMsystem), exact segmentation represents a crucial step for finalclassification, a concept was presented which realizes as exact asegmentation as possible of white blood cells also on images of bonemarrow smears.

Embodiments of the present invention describe a method of segmentingleukocytes in microscopic pictures by way of example of bone marrowsmears. In contrast to known methods (as were set forth in theintroductory part of this application), the following problems aresolved by the approach presented here: robust detection of the differentcell types, robust segmentation of the different cell types, or robustsegmentation of the different cell types in cell clusters.

Segmentation of leukocytes, which was described in connection with themethod 200, is an important component of a CAM system mentioned above.It represents the initial step of such a system, wherein the individualcell is initially to be detected and, subsequently, to be segmented withas much precision as possible. Exact segmentation has a crucial effectin particular on the subsequent step, the classification of the cell.The individual steps of the method 200 of segmenting leukocytes wereexplained in more detail by way of example of pictures of bone marrowsmears, and the course of the method 200 was shown by means of theflowchart in FIG. 2. Embodiments of the present invention (i.e., forexample, the method 200) expect, as an input, an image containing one ormore leukocyte clusters. Common picture formats are, e.g., RGB imageshaving a depth of 8 bits per channel (per color channel) and pixel. Ithas been found that a 100-fold magnification is a fair compromisebetween the number of mapped cells in a picture and a sufficient degreeof detail so as to be able to recognize the boundaries between twocells. One aim of embodiments of the present invention is to localizeand segment any leukocytes present. The output of embodiments of thepresent invention is a collection of regions, each region ideallycorresponding to precisely one leukocyte.

Embodiments of the present invention thus enable robust detection andsegmentation of leukocytes in bone marrow smears. In contrast to knownmethods (as were described, for example, in the introductory part ofthis application), over-segmentation is minimal. Embodiments of thepresent invention will still provide good results even if the images aresoiled by burst cells, dirt and other artefacts that occasionally resultfrom taking the samples. In addition, variations in the illumination orthe background color may be compensated for, and will not or onlyminimally disturb detection and segmentation.

Even though some aspects were described in connection with an apparatus,it is understood that said aspects also represent a description of thecorresponding method, so that a block or a component of an apparatus isalso to be understood as a corresponding method step or as a feature ofa method step. By analogy therewith, aspects that were described inconnection with or as a method step also represent a description of acorresponding block or detail or feature of a corresponding apparatus.Some or all of the method steps may be performed by a hardware apparatus(or while using a hardware apparatus) such as a microprocessor, aprogrammable computer or an electronic circuit. In some embodiments,some or several of the most important method steps may be performed bysuch an apparatus.

Depending on specific implementation requirements, embodiments of theinvention may be implemented in hardware or in software. Implementationmay be performed using a digital storage medium, for example a floppydisc, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM,or a flash memory, a hard disc or any other magnetic or optical memorywhich has electronically readable control signals stored thereon thatmay cooperate, or indeed do cooperate, with a programmable computersystem such that the respective method is performed. This is why thedigital storage medium may be computer-readable.

Some embodiments in accordance with the invention thus include a datacarrier having electronically readable control signals that are capableof cooperating with a programmable computer system such that any of themethods described herein is performed.

Generally, embodiments of the present invention may be implemented as acomputer program product having a program code, the program code beingoperative to perform any of the methods when the computer programproduct runs on a computer.

The program code may also be stored on a machine-readable carrier, forexample.

Other embodiments include the computer program for performing any of themethods described herein, the computer program being stored on amachine-readable carrier.

In other words, an embodiment of the inventive method thus is a computerprogram having a program code for performing any of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods thus is a data carrier (ora digital storage medium or a computer-readable medium) on which thecomputer program for performing any of the methods described herein isrecorded.

A further embodiment of the inventive method thus is a data stream or asequence of signals representing the computer program for performing anyof the methods described herein. The data stream or the sequence of datamay be configured, e.g., to be transferred via a data communicationlink, for example via the interne.

A further embodiment includes a processing means, for example a computeror a programmable logic device, configured or adapted to perform any ofthe methods described herein.

A further embodiment includes a computer on which the computer programfor performing any of the methods described herein is installed.

In some embodiments, a programmable logic device (e.g. afield-programmable gate array, an FPGA) may be used for performing someor all of the functionalities of the methods described herein. In someembodiments, a field-programmable gate array may cooperate with amicroprocessor to perform any of the methods described herein. In someembodiments, the methods are generally performed by any hardware device.The latter may be a universally employable hardware such as a computerprocessor (CPU) or a hardware specific to the method, such as an ASIC,for example.

The above-described embodiments merely represent an illustration of theprinciples of the present invention. It is to be understood thatmodifications and variations of the arrangements and details describedherein will be appreciated by other persons skilled in the art. This iswhy it is intended that the invention be limited only by the scope ofthe following claims rather than by the specific details that werepresented herein by means of the description and the explanation of theembodiments.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

The invention claimed is:
 1. A method of segmenting biological cells ina picture, so that the biological cells represent a foreground of thepicture, the method comprising: applying a first fast marching algorithmto the picture or to a pre-processed version of same so as to achieve afirst fast marching image, the first fast marching algorithm startingfrom a background of the picture, and a velocity function of the firstfast marching algorithm being based on a first edge-strength image ofthe picture; segmenting the first fast marching image or afurther-processed version of same into a plurality of homogeneousregions; mapping each of the homogeneous regions to one node,respectively, of a graph so that nodes of adjacent homogeneous regionsare connected to one another and so that the graph comprises roots whichcorrespond to homogeneous regions located at cell centers; classifying,on the basis of the graph, each homogeneous region either as backgroundor as foreground; and applying a second fast marching algorithm withinthe homogeneous regions classified as foreground to a secondedge-strength image so as to segment the foreground into individualbiological cells, the second fast marching algorithm starting from thosehomogeneous regions which correspond to the roots of the graph.
 2. Themethod as claimed in claim 1, wherein in the application of the firstfast marching algorithm, the velocity function of the first fastmarching algorithm is further based on a distance function indicating,for each pixel of the picture, a distance of a color value of therespective pixel from a cell estimation color value associated with therespective pixel.
 3. The method as claimed in claim 1, furthercomprising determining, by means of a color threshold-value method,pixels of the picture which are very likely to belong to the foregroundof the picture; and further comprising associating a cell estimationcolor value corresponding to a central tendency of samples of pixels ina predetermined environment of the respective pixel, which are verylikely to belong to the foreground, with each pixel that is not verylikely to belong to the foreground of the picture.
 4. The method asclaimed in claim 1, further comprising determining pixels that are verylikely to belong to a foreground of the picture, and of determiningpixels that are very likely to belong to a background of the picturewhile using a threshold-value method; and wherein in the application ofthe first fast marching algorithm, the first fast marching algorithmstarts from the pixels of the picture which are very likely to belocated in the background of the picture.
 5. The method as claimed inclaim 4, wherein during determining pixels that are very likely tobelong to a foreground of the picture and/or pixels that are very likelyto belong to a background of the picture, the picture or afurther-processed version of same is transformed to a gray-level imagesuch that a likelihood of belonging to the foreground continuouslyincreases or decreases along a gray-level scale of the gray-level imageso as to determine, on the basis of the first threshold value of thethreshold-value method, those pixels that are very likely to belong tothe foreground and to subsequently determine, on the basis of a secondthreshold value of the threshold-value method, from a quantity of pixelsthat are very likely to not belong to the foreground, those pixels thatare very likely to belong to the background.
 6. The method as claimed inclaim 1, further comprising determining the edge-strength image, whereineach pixel of the picture has assigned to it an edge-strength valuewhich depends on a gradient of a color curve of the picture or of thefurther-processed version of same at the respective pixel.
 7. The methodas claimed in claim 1, wherein the velocity function of the first fastmarching algorithm is calculated for a pixel in accordance with${{F_{bg}( {x,y} )} = \frac{{D( {x,y} )}^{\alpha} + \theta}{\gamma + {I_{grad}( {x,y} )}}},$wherein (x,y) indicates a coordinate position of the pixel within thepicture, F_(bg)(x,y) indicates a velocity of the first fast marchingalgorithm in the pixel, D(x,y) indicates a distance of a color value ofthe pixel from a cell estimation color value associated with the pixel,I_(grad)(x,y) indicates an edge-strength value of the pixel, and α, θ, γare predetermined constants.
 8. The method as claimed in claim 1,wherein segmenting of the first fast marching image or of thefurther-processed version of same comprises combining, in a hierarchicalmanner, pixel clusters of pixels of the first fast marching image or ofthe further-processed version of same which are mutually adjacent andcomprise a similar color value.
 9. The method as claimed in claim 1,wherein in the mapping of the homogeneous regions to one node,respectively, of the graph, an order of a homogeneous region within thegraph is based on a sample of the homogeneous region, a sample of ahomogeneous region being a mean value of samples of pixels associatedwith the homogeneous region.
 10. The method as claimed in claim 1,wherein a homogeneous region is located at a cell center if a node towhich the homogeneous region is mapped is a root of the graph; and ifthe homogeneous region was classified as foreground duringclassification.
 11. The method as claimed in claim 1, wherein duringclassifying, a homogeneous region is classified as background, if asample of the homogeneous region is smaller than a first predefinedvalue; or if a node to which the homogeneous region is mapped forms aleaf of the graph; or if a length of a longest path, in an ascendingdirection with regard to the order of the graph, from a leaf of thegraph to a root of the graph through the node to which the homogeneousregion is mapped is smaller than a second predefined value; or if thecolor value of the homogeneous region is smaller than a third predefinedvalue, a length of a shortest path, in a direction that is descending interms of the order of the graph, from the node to which the homogeneousregion is mapped to a closest leaf of the graph is smaller than a fourthpredefined value, and if more than one root of the graph may be reached,starting from the node to which the homogeneous region is mapped, in theascending direction.
 12. The method as claimed in claim 1, wherein avelocity function of the second fast marching algorithm adopts the valuezero for any homogeneous regions classified as background.
 13. Themethod as claimed in claim 1, wherein the second fast marching algorithmis configured such that a meeting line between a first fast marchingfront, which starts from a first cell center of a first biological cell,and a second fast marching front, which starts from a second cell centerof a second biological cell, forms a boundary between the firstbiological cell and the second biological cell.
 14. The method asclaimed in claim 1, wherein the first edge-strength image is identicalwith the second edge-strength image.
 15. An apparatus for segmentingbiological cells in a picture, so that the biological cells represent aforeground of the picture, the apparatus comprising: a fast marchingprocessor configured to apply a first fast marching algorithm to apicture or a pre-processed version of same so as to achieve a first fastmarching image, the first fast marching algorithm starting from abackground of the picture, and a velocity function of the first fastmarching algorithm being based on a first edge-strength image of thepicture; a segmenter configured to segment the first fast marching imageor a further-processed version of same into a plurality of homogeneousregions; a mapper configured to map each of the homogeneous regions toone node, respectively, of a graph so that nodes of adjacent homogeneousregions are interconnected and that the graph comprises roots whichcorrespond to homogeneous regions located at cell centers; and aclassifier configured to classify each homogeneous region either asbackground or as foreground on the basis of the graph; said fastmarching processor being further configured to apply a second fastmarching algorithm within the homogeneous regions classified asforeground to a second edge-strength image so as to segment theforeground into individual biological cells, said fast marchingprocessor being configured such that said second fast marching algorithmstarts from those homogeneous regions that correspond to the roots ofthe graph.
 16. A non-transitory computer-readable medium having storedthereon a computer program comprising a program code for performing themethod of segmenting biological cells in a picture, so that thebiological cells represent a foreground of the picture, the methodcomprising: applying a first fast marching algorithm to the picture orto a pre-processed version of same so as to achieve a first fastmarching image, the first fast marching algorithm starting from abackground of the picture, and a velocity function of the first fastmarching algorithm being based on a first edge-strength image of thepicture; segmenting the first fast marching image or a further-processedversion of same into a plurality of homogeneous regions; mapping each ofthe homogeneous regions to one node, respectively, of a graph so thatnodes of adjacent homogeneous regions are connected to one another andso that the graph comprises roots which correspond to homogeneousregions located at cell centers; classifying, on the basis of the graph,each homogeneous region either as background or as foreground; andapplying a second fast marching algorithm within the homogeneous regionsclassified as foreground to a second edge-strength image so as tosegment the foreground into individual biological cells, the second fastmarching algorithm starting from those homogeneous regions whichcorrespond to the roots of the graph, when the program runs on acomputer.